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Abstract 

This paper provides some universal information-theoretic bounds related to capacity-approaching ensembles of low-density 
parity-check (LDPC) codes. These bounds refer to the behavior of the degree distributions of such ensembles, and also 
' to the graphical complexity and the fundamental system of cycles associated with the Tanner graphs of LDPC ensembles. 
. The transmission of these ensembles is assumed to take place over an arbitrary memoryless binary-input output-symmetric 
(MBIOS) channel. The universality of the bounds derived in this paper stems from the fact that they do not depend on the 
full characterization of the LDPC ensembles but rather depend on the achievable gap between the channel capacity and the 
"q ' design rate of the ensemble, and also on the required bit error (or erasure) probability at the end of the decoding process. 
. Some of these bounds hold under maximum-likelihood decoding (and hence, they also hold under any sub-optimal decoding 
algorithm) whereas the others hold particularly under the sum-product iterative decoding algorithm. The tightness of some of 
these bounds is exemplified numerically for capacity-approaching LDPC ensembles under sum-product decoding; the bounds 
are reasonably tight for general MBIOS channels, and are tightened for the binary erasure channel. 
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>- ■ I. Introduction 

Low-density parity-check (LDPC) codes were introduced by Gallager [7] in the early 1960s. These linear block 
. codes are characterized by sparse parity-check matrices which facilitate their efficient decoding with sub-optimal 
O \ iterative message-passing algorithms. In spite of the seminal work of Gallager [7], LDPC codes were ignored for 
Q\ • a long time. Following the breakthrough in coding theory made by the introduction of turbo codes [2] and the 
^ . rediscovery of LDPC codes [13] in the mid 1990s, it was realized that these codes and lots of other variants of 
q [ capacity-approaching error-correcting codes can all be understood as codes defined on graphs. Graphs not only 
^ ■ describe the codes, but more importantly, they structure the operation of efficient iterative decoding algorithms 
which are used to decode these codes. Various iterative algorithms, used to decode codes defined on graphs, enable 
to closely approach the channel capacity while maintaining reasonable decoding complexity. This breakthrough 
■ attracted coding theorists, and lots of research activity has been conducted during the last decade on these modern 
coding techniques and their practical decoding algorithms; the reader is referred to the special issue of the IEEE 
Transactions on Information Theory on codes on graphs and iterative algorithms [26]. 

This paper derives some universal information-theoretic bounds related to capacity-approaching LDPC ensembles 
whose transmission takes place over a memoryless binary-input output-symmetric (MBIOS) channel. The universal- 
ity of the bounds derived in this paper stems from the fact that they do not depend on the full characterization of the 
LDPC ensembles but rather depend on the achievable gap between the channel capacity and the design rate of the 
ensemble, and also on the required bit error (or erasure) probability at the end of the decoding process. Most of these 
bounds apply to the asymptotic case where we let the block length tend to infinity, and the bounds are compared 
with some capacity-approaching LDPC ensembles. The design of such ensembles under iterative decoding lies on 
a solid background due to the density evolution technique which was developed by Richardson and Urbanke (see 
[10], [18], [19]). This technique is commonly used for a numerical search of the degree distributions of capacity- 
approaching LDPC ensembles where the target is to minimize the gap to capacity for infinite block length while 
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limiting the maximal degree and specifying the communication channel model. Some approximate techniques which 
optimize the degree distributions of LDPC ensembles under further practical constraints (e.g., an optimization of the 
degree distributions of LDPC ensembles for obtaining a good tradeoff between the asymptotic gap to capacity and 
the decoding complexity [1]) are of interest. For the binary erasure channel (BEC), the density evolution technique 
is tractable for analysis (as it becomes a one-dimensional analysis), and explicit expressions for capacity-achieving 
sequences of LDPC ensembles for the BEC are introduced in [12], [17], [24]. For general MBIOS channels, as 
of yet there are no closed-form expressions for capacity-achieving sequences of LDPC ensembles under iterative 
decoding, and the density evolution technique is used as a numerical tool for devising the degree distributions of 
capacity-approaching LDPC ensembles in the limit where their block length tends to infinity. 

It is well known that linear block codes which are represented by cycle-free Tanner graphs have poor performance 
even under ML decoding [6]. The Tanner graphs of capacity-approaching LDPC codes should have cycles, and 
it is of interest to provide some quantitative bounds related to the average cardinality of these cycles for LDPC 
ensembles. Following the approach of Khandekar and McEliece in [11] where the decoding complexity is measured 
in terms of the achievable gap (in rate) to capacity, we also study in this paper the behavior of the degree distributions 
and the average cardinality of the fundamental system of cycles of LDPC ensembles in terms of their achievable 
gap to capacity. Several of the results derived in this paper hold under ML decoding (and, hence, they also hold 
under any sub-optimal decoding algorithm), while some other results are specialized to the iterative sum-product 
decoding algorithm. 

This paper is structured as follows: Section|irjpresents preliminary background and notation, SectionlTTllintroduces 
the main results of this paper, Section [TV] provides their proofs followed by some discussions, and Section [V] 
exemplifies the numerical tightness of some of the bounds derived in this paper. Finally, Section [VTJ summarizes 
this paper, and provides some open problems which are related to this research. 

II. Preliminaries 

We introduce here some background from [8], [21], [25] and [28] and notation which serve for the rest of this 
paper. 



A. LDPC Ensembles 



LDPC codes are linear block codes which are characterized by sparse parity-check matrices. Alternatively, a 
parity-check matrix can be represented by a bipartite (Tanner) graph where the variable and parity-check nodes 
which specify the binary linear block code are on the left and the right of this graph, respectively, and an edge 
connects between a variable node and a parity-check node if the corresponding code symbol is involved in the 
specific parity-check equation. This is illustrated in Fig. [T] 
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Fig. 1. A parity-check matrix H and the corresponding Tanner graph. For illustrating this relationship, column 8 and row 2 of H are 
emphasized; the corresponding variable and parity-check nodes, and the attached edges are also emphasized (this figure appears in [20], and 
it is used later in this section as a reference). 



We now move to consider ensembles of LDPC codes. The requirement of the sparseness of the parity-check 
matrices representing LDPC codes is transformed to an equivalent requirement where the number of edges in the 
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Tanner graph scales linearly with the block length (note that by picking a parity-check matrix uniformly at random 
as a representative of a binary linear block codes, the number of non-zero elements in this matrix is likely to scale 
quadratically with the block length). Following standard notation in [21], let Aj and pi denote the fraction of edges 
attached to variable and parity-check nodes of degree i, respectively. In a similar manner, let A, and Tj denote the 
fraction of variable and parity-check nodes of degree i, respectively. The LDPC ensemble is characterized by a 
triplet (n, A, p) where n designates the block length of the codes, and the power series 



represent, respectively, the left and right degree distributions from the edge perspective. Equivalently, this ensemble 
is also characterized by the triplet (n, A, V) where the power series 

oo oo 

A(*)^J> 4 x\ r(*)^r,x* 

i=l i=l 

represent, respectively, the left and right degree distributions from the node perspective. We denote by LDPC(n, A, p) 
(or LDPC(n, A, T)) the ensemble of codes whose bipartite graphs are constructed according to the corresponding 
pairs of degree distributions. The connections between the edges emanating from the variable nodes to the parity- 
check nodes are constructed by numbering the connectors on the left and the right of the graph (whose number 
is the same on both sides, and is equal to s = n YliLi i^iX an d by using a random and uniform permutation 
7T : {1, . . . , s} — > {1, . . . , s} which associates connector number i (where 1 < i < s) on the left side of this graph 
with the connector whose number is ir(i) on the right. One can switch between degree distributions w.r.t. to the 
nodes and edges of a bipartite graph, using the following equations: 



px px 

/ \(u)du I p(u)d 

A(x) = ^ , r(x) - 



\(u)du I p(u)du 

o Jo 

A'(x) T'(x) 

An important characteristic of an ensemble of LPDC codes is its design rate. For an LDPC ensemble whose codes 
are represented by parity-check matrices of dimension c x n, the design rate is defined to be R d = 1 — ^. This 
serves as a lower bound on the actual rate of any code from this ensemble, and the rate of such a code is equal to 
the design rate if the particular parity-check matrix representing this code is full rank. For an ensemble of LDPC 
codes, the design rate is given in terms of the degree distributions (either w.r.t. the edges or nodes of a Tanner 
graph), and it can be expressed in two equivalent forms: 

/ p(x)dx , m 

/ \{x)dx 
Jo 

Note that 

«L = A'(l) = — ^ , a R = r'(i) = 



/ X(x)dx / p(x)dx 

Jo Jo 



designate the average left and right degrees, respectively (i.e., the average degrees of the variable nodes and parity- 
check nodes, respectively). 

In this paper, we rely on the stability condition which forms a necessary condition for a successful iterative 
message -passing decoding for LDPC ensembles. The reader is referred to [21, Chapter 4] for more background on 
the analytical tools used for the asymptotic analysis of LDPC ensembles over MBIOS channels, and the stability 
condition in particular. 
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B. Elements from Graph Theory 

Definition 1: [Tree] A tree is a connected graph that has no cycles. 
From DennitiondJ trees are the smallest connected graphs; remove any edge from a tree and it becomes disconnected. 
A fundamental property of trees is that any two vertices are connected by a unique path. 

Every graph Q has subgraphs that are trees. This motivates the following definition: 

Definition 2: [Spanning tree] A spanning tree of a graph Q is a tree which spans all the vertices of Q. 

Lemma 1: Every connected graph has a spanning tree. 

Proof: Let Q be a connected graph. If Q has no cycles, then it is a spanning tree. Otherwise, choose a cycle 
S in the graph, and remove an arbitrary edge e which belongs to this cycle. The remaining graph is still connected 
since any path which uses the removed edge e can now be replaced by a path using S — {e}, so that every two 
vertices of the graph Q are still connected. One can repeat this process as many times as required until the resulting 
graph has no cycles. By construction, it is a spanning tree of Q. ■ 

Definition 3: [Fundamental cycle of a connected graph] Let J 7 be a spanning tree of a graph Q, and let e be 
any edge in the relative complement of T. The cycle of the subgraph JFU {e} (whose existence and uniqueness is 
guaranteed by [8, Theorem 3.1.11]) is called a fundamental cycle of Q which is associated with the spanning tree 
T. 

We turn now to discuss graphs with a finite number of components. 

Definition 4: [Number of components of a graph] Let Q be a graph (possibly disconnected). The number of 
components of Q is the minimal number of its connected subgraphs whose union forms the graph Q (clearly, a 
connected graph has a unique component). 

Definition 5: [Cycle rank] Let Q be an arbitrary graph with \Vg\ vertices, \Eg\ edges and C(Q) components. 
The cycle rank of Q, denoted by [3(G), equals the maximal number of edges which can be removed from the graph 
without increasing its number of components. 

From Definition |51 the cycle rank of a graph is a measure of the edge redundancy with respect to the connectedness 
of this graph. The cycle rank satisfies the following equality (see [8, p. 154]): 

0{G) = \Eg\-\Vg\ + C(g). (2) 

Definition 6: [Full spanning forest] Let Q be an arbitrary graph. The full spanning forest T of the graph Q is 
the subgraph of Q after removing the (3(G) edges from Definition [5] Clearly, the number of components of T and 
Q is the same. 

Definition 7: [Fundamental cycle] Let T be a full spanning forest of a graph Q, and let e be any edge in the 
relative complement of T . The cycle of the subgraph T U {e} (whose existence and uniqueness is guaranteed by 
[8, Theorem 3.1.11]) is called a fundamental cycle of Q which is associated with T. 

Remark 1: Each of the edges in the relative complement of a full spanning forest T gives rise to a different 
fundamental cycle of the graph Q. 

Definition 8: [Fundamental system of cycles] The fundamental system of cycles of a graph Q which is associated 
with a full spanning forest T is the set of all fundamental cycles of Q associated with T. 

Remark 2: By Remark [T] the cardinality of the fundamental system of cycles of Q associated with a full spanning 
forest of this graph is equal to the cycle rank (3(G)- 

Example 1: [Illustration with the Tanner graph in Fig. [2 We refer now to the Tanner graph in Fig. [T] This 
graph is connected, but it is clearly not a tree. As an example of a cycle in this graph, we refer to the path which 
connects vg to C4, C4 to v\o, i>io to C5, and finally C5 back to vg; this cycle includes 4 edges, so it is a cycle of 
the shortest length. Since the number of vertices in this graph is 15 and the number of edges is 30, then from (0, 
the cycle rank of this connected Tanner graph is equal to 30 — 15 + 1 = 16. In order to get a spanning tree of this 
graph, we follow the concept of the proof of Lemma [T] and remove sequentially 16 suitable edges from this graph 
so that it still remains connected. In the following, we show the parity-check matrix H which corresponds to the 
new Tanner graph after the removal of the above 16 edges from the original graph in Fig. [TJ the new zero entries 
of H which correspond to these removed edges are bolded. 

The matrix H with its bolded zeros defines a set of 16 fundamental cycles of the Tanner graph in Fig. [T] For 
example, by returning the edge which connects vq with c\ (i.e., returning the value in the first row and sixth column 
of H to be 1), we get a fundamental cycle; this is the path which connects V3 to C2, C2 to vq, vq to c\, and c\ back 
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123456789 10 

/1110000000 \ 
110 1110 
0000000010 
1000100000 
\100000001 1 / 

Fig. 2. A parity-check matrix which corresponds to a spanning tree of the Tanner graph in Fig. [JJ As compared to the parity-check matrix 
H in Fig.Q] the new parity-check matrix H is obtained by changing the values of the emphasized 16 entries from 1 to 0. 

to vs). This is clearly a fundamental cycle in this graph since it is a cycle of the shortest possible length. The new 
Tanner graph which corresponds to H is connected. For example, the variable nodes v$ and vq are connected by 
the path of length 6 which connects vq to Q, C2 to v%, V3 to c\, c\ to v\, v\ to C4 and C4 to V5. This path can be 
observed directly from the parity-check matrix H = [hi j] by alternate horizontal and vertical moves through the 
ones of H; explicitly, this path is determined by the horizontal move from /i2,6 to /i2,3, then the vertical move to 
/113, the horizontal move to h\\, the vertical move to /141 and the horizontal move to /14 5. In a similar way one 
can show that each two vertices of the Tanner graph which corresponds to H are connected, and hence this graph 
is indeed a spanning tree of the Tanner graph in Fig. [T] Every single edge which is added to the new graph creates 
a fundamental cycle (like the one mentioned above). 

C. Lower Bound on the Conditional Entropy for Binary Linear Block Codes 

In this section, we outline the derivation of a lower bound on the conditional entropy of a transmitted codeword 
given the received sequence at the output of the channel; the full derivation is given in [25, Section 4] and its 
appendices, and it is outlined here in order to highlight the main steps used for this derivation. The lower bound 
on the conditional entropy which is presented in this section is crucial for the proof of Theorem Q] of this paper. 

In the sequel to this discussion, we assume that the transmission of a binary linear block code takes place over 
an MBIOS channel. In the following, the block length and the code rate are designated by n and R, respectively. 
Let C designate the capacity of this communication channel in units of bits per channel use. 

• Define an equivalent channel whose output is the log-likelihood ratio (LLR) of the original communication 
channel. 

• The LLR is represented by a pair which includes its sign and absolute value. 

• For the characterization of the equivalent channel, let the function a designate the conditional pdf of the LLR 
given that the channel input is the zero symbol. 

• We randomly generate an i.i.d. sequence 1 w.r.t. the conditional pdf a, and define 

if Li > 

1 if Li < . 
or 1 w.p. 5 if Li = 

• The output of the equivalent channel is Y = {Y\ , . . . , Y n ) where 

Yi = ($i,tti), i = l,...,n 

and = 0j + Xj (this addition is modulo-2). 

• The output of this equivalent channel at time i is therefore the pair Oj) where <2>j G {0, 1} and Qj € M + . 
This defines the memoryless mapping 

X -»■ Y = 

where $ is a binary random variable which is affected by X, and Q, is a non-negative random variable which 
is not affected by X. 
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Due to the symmetry of the communication channel, the pdf of the absolute value of the LLR satisfies 

' a(u) + a(-uj) = (1 + e _a; ) a(u) if lo > 0, 
a(0) if u = 0. 



Let C be a binary linear block code of length n and rate R. In addition, let X and Y be the transmitted codeword and 
received sequence, respectively. The conditional entropy of the transmitted codeword given the received sequence 
at the output of the MBIOS channel satisfies 

#(X|Y) = #(X|Y) 

= JT(X) + H(Y\X) - H(Y) 
= nR + nH^Xi) - H(Y) 

= nR + n[H(Y 1 )-I(X 1 ;Y 1 )]-H(Y) (3) 

and 

I(X l] Y l ) = I(X l] Y l )<C (4) 

H(Y) = 

= H(Q)+H($\n) 

= H(Q) + 1. (5) 

The last transition in ((5]) is due to the fact that given the absolute value of the LLR, its sign is equally likely to be 
positive or negative. The entropy H(Q) is not expressed explicitly as it will cancel out later. 
The entropy of the vector Y satisfies 

H(Y) = H(<s> 1 ,n 1 ,...,<s> n ,n n ) 

= H(n u ...,n n ) + #■(*!, . . . , $ n | Oi, . . . , n n ) 

= nH(n) + H(^ 1 ,...,^ n | nt,...,n n ). (6) 

• Define the syndrome vector S = (<&i, . . . , & n )H T where H is an arbitrary full-rank parity-check matrix of C. 

• Let M be the index of the vector ($1, . . . , <6 n ) in the coset. 

• H(M) = nR since all the codewords are transmitted with equal probability, and we get 

(($!,..., $ n ) | (fia,...,^)) 
= H{S,M\(Q 1 ,...,Q n )) 

<H(M) + H(s\(n 1 ,...,n n )) 

n(l-fi) 

<nR+ H(S j \(n 1 ,...,n n )) . (7) 

• Since Xif T = for any codeword X, and <3?j = X{ + @i for all i, then S = (81, . . . , Q n )H T which is 
independent of the transmitted codeword. 

By combining (H)-©, one obtains that 

n(l-R) 

ff(X| Y) > n(l - C) - fl'(5 i |J2i > ...,n n ) 

J'=l 

where 

• Sj = 1 if and only if 0j = 1 for an odd number of i's in the j'th parity-check equation. 

• Due to the symmetry of the channel 

P(oi) = P(e i = = at) = 
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In order to calculate the conditional entropy of a single component of the syndrome, the following lemma is used: 
Lemma 2: If the j'th component of the syndrome S involves k active variables whose indices are ... ,if.} 
then 

1 



P(Sj = 1 = ax, . . . ,Qi 



Oik) 



i - n i 1 - 2p («-)) 



m=l 



where 



1 - 2P(a) = tanh 



a 



For a parity-check equation of degree k, the conditional entropy H(Sj fii, . . . , Q n ) is given by the following 
^-dimensional integral: 




1 - JJ tanh(-^) J Y\ /n( a m) d«i • • • da k 

m=l / m=l 

where designates the pdf of the absolute value of the LLR, and /i 2 designates the binary entropy function 
to the base 2. 

• Using the Taylor series expansion of /12 (around one-half) transforms the above multi-dimensional integral to 
a one-dimensional integral raised to the fc'th power. 

For an arbitrary full-rank parity-check matrix of a binary linear block code C, let Tj designate the fraction of the 
parity-checks involving i variables, and let T(x) = 2~2i^i xl - This leads in [25, Eq. (56)] to the following lower 
bound on the conditional entropy of the transmitted codeword given the received sequence at the channel output: 



#(X|Y) 



> R-C + 



1 



R ^ 



n 



2 In 2 ^k(2k 



where 



9k = a(/)(l + e-')tanh 2fc Q dl, k G N. 



(8) 



(9) 



The above lower bound on the conditional entropy holds for any representation of the code by a full-rank parity- 
check matrix. Note also that the symmetry condition states that a(l) = e l a(—l) for all / G M, and therefore (|9]) 
gives that 



9k 



E 



k G N 



(10) 



where E designates the statistical expectation, and L is a random variable which stands for the LLR at the output 
of the channel given that the input bit is zero. This also implies that the sequence {g^} depends only on the 
communication channel (and not on the code). 

We note that for a general MBIOS channel, the lower bound on the conditional entropy which is given in © 
is tighter than the bound in [3, Eq. (15)]. These two bounds coincide however for the case of a binary symmetric 
channel (BSC) since the derivation of the bound in ([8]) relies on the soft output of the channel whereas the derivation 
of the bound in [3, Eq. (15)] relies on a two-level quantization of this output (which turns the side information 
about the LLR at the output of the MBIOS channel to be equivalent to the side information which is obtained from 
a degraded BSC, unless this channel was originally a BSC where the two bounds then coincide). 



D. Notation 

We rely on the following standard notation (see [28]): 

• f(n) = 0[g(n)) means that there are positive constants c and k, such that < f(n) < c g(n) for all n > k. 
The values of c and k must be fixed for the function / and should not depend on n. 

• /(n) = f2((?(n)) means that there are positive constants c and k, such that < c g{n) < f(n) for all n> k. 
The values of c and k must be fixed for the function / and should not depend on n. 

In this paper, we refer to the gap (in rate) to capacity, denoted by e, and discuss in particular the case where 
< e <^ 1 (i.e., capacity-approaching ensembles). Accordingly, we have 
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• /(e) = O (17(e)) means that there are positive constants c and 5, such that < /(e) < c g(e) for all < e < 5. 
The values of c and 5 must be fixed for the function / and should not depend on e. 

• /(e) = Q (17(e)) means that there are positive constants c and S, such that < c 17(e) < /(e) for all < e < 5. 
The values of c and 5 must be fixed for the function / and should not depend on e. 

Throughout the paper 

^2(2^) = —x log 2 (x) — (1 — x) log 2 (l — x), < x < 1 
designates the binary entropy function to the base 2, and /i 2 1 : [0, 1] — > [0, ^] stands for its inverse function. 

III. Main Results 

The following theorem provides an information-theoretic lower bound on the average degree of the parity-check 
nodes of an arbitrary Tanner graph representing a binary linear block code; it is assumed that the graph corresponds 
to a full-rank parity-check matrix of this code. The new bound forms a tightened version of a previously reported 
lower bound (see [25, Eq. (77)]). We later generalize this theorem for LDPC ensembles. 

Theorem 1: [On the average degree of the parity-check nodes] Let C be a binary linear block code whose 
transmission takes place over an MBIOS channel. Let Q be a standard bipartite (Tanner) graph which represents 
the code while referring to a full-rank parity-check matrix of the code, let C designate the channel capacity in units 
of bits per channel use, and a be the conditional pdf of the log-likelihood ratio (LLR) at the output of the channel 
given that the input is zero. Let the code rate be (at least) a fraction 1 — e of the channel capacity (where e G (0, 1) 
is arbitrary), and assume that this code achieves a bit error probability P\> under some decoding algorithm. Then 
the average right degree of the Tanner graph of the code (i.e., the average degree of the parity-check nodes in Q) 
satisfies 




l-2fe- 1 ' 



In 

where g\ only depends on the channel and is given by 



9i 



(11) 



A 

91 = 



J a{l)(l + e~ / )tanh 2 (^\ dl. (12) 



For the BEC, this bound is tightened to 



ln(l + 
ln 



V-Pb 



. (l-p)e+fly 

or > — r^~\ (I 3 ) 



where p is the erasure probability of the BEC and P\, is the erasure probability of the code. Furthermore, among all 
the MBIOS channels with a fixed capacity C, and for fixed values of the gap (in rate) to capacity (e) and bit error/ 
erasure probability (Pb), the lower bound on the average degree of the parity-check nodes given in (TTTb attains its 
maximal and minimal values for a BSC and BEC, respectively. 

Remark 3: In the particular case where Pb vanishes, the lower bound on the average right degree introduced in 
(fTTb forms a tightened version of the bound given in [25, Eq. (77)]; note also that the bit error probability is taken 
into account in Theorem Q] while it is assumed to vanish in [25, Eq. (77)]. This point and some of its implications 
are clarified in Discussion [T] which proceeds the proof of Theorem Q] In the limit where the gap (in rate) to capacity 
vanishes, the lower bounds on the average right degree in (fTTb and [25, Eq. (77)] both grow like the logarithm of 
the inverse of the gap to capacity and they possess the same behavior. In this case 

a R = n (in . (14) 

Besides of tightening the bound in [25, Eq. (77)] and generalizing it to the case where the bit error (or erasure) 
probability is strictly positive, Theorem [TJ also states the two extreme cases among all MBIOS channels with fixed 
capacity. 
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Remark 4: As is clarified later in Discussion |2l Theorem Q] can be adapted to hold for an arbitrary ensemble of 
(n, A, p) LDPC codes. In this case, the requirement of a full-rank parity-check matrix of a particular code C from 
this ensemble is relaxed by requiring that the design rate of the LDPC ensemble forms a fraction 1 — e of the 
channel capacity. In this case, P\, stands for the average bit error (or erasure) probability of the ensemble under 
some decoding algorithm. 

Corollary 1: Under the assumptions in Theorem [TJ the cardinality of the fundamental system of cycles of a 
Tanner graph Q, associated with a full spanning forest of this graph, is larger than 



n 



/2)(o R - 1) - 1] (15) 

where from Theorem \T\ or can be replaced by the lower bounds in (ITTb and (|T3T > for a general MBIOS channel 
and a BEC, respectively. From this corollary and Remark |3j the cardinality of the fundamental system of cycles of 
the Tanner graph Q which is associated with a full spanning forest of this graph is (in i) . 
Based on Remark @] and Corollary [T] the following result is derived: 

Corollary 2: [On the asymptotic average cardinality of the fundamental system of cycles of LDPC ensem- 
bles] Consider a sequence of LDPC ensembles, specified by an arbitrary pair of degree distributions (X,p), whose 
transmission takes place over an MBIOS channel. Let the design rate of these ensembles be a fraction 1 — e of the 
channel capacity C (in units of bits per channel use), and assume that the average bit error/ erasure probability of 
such an LDPC ensemble vanishes under some decoding algorithm as we let the block length tend to infinity. Then, 
the average cardinality of the fundamental system of cycles of a Tanner graph Q (denoted by (3(G)) satisfies the 
following asymptotic property when the average is taken over all the Tanner graphs which represent codes from 
an LDPC ensemble of this sequence and as we let the block length tend to infinity: 

^ldpc( n,A,p) [(3(G)] 



lim inf ■ 

n— >oc n 

(l-C) InL ' 

> " 1 TT^- — — " 1 < 16 ) 

lnf 1 



J 1 . 

where g\ in introduced in (fl2l ). For a BEC whose erasure probability is p, a tightened version of this result gets 
the form 

,„(_!_) 

Furthermore, among all the MBIOS channels with a fixed capacity C and for a fixed value of the achievable gap 
in rate (e) to the channel capacity, the lower bound (fl6l ) on the asymptotic average cardinality of the fundamental 
system of cycles attains its maximal and minimal values for a BSC and BEC, respectively. 
Remark 5: Corollary [2] provides two results which are of the type O (ln^). 

Theorem 2: [On the degree distributions of capacity-approaching LDPC ensembles] Let (n, A, p) be an 

ensemble of LDPC codes whose transmission takes place over an MBIOS channel. Assume that the design rate of 
the ensemble is equal to a fraction 1 — e of the channel capacity C, and let P\, designate the average bit error (or 
erasure) probability of the ensemble under ML decoding or any sub-optimal decoding algorithm. Then, the following 
properties hold for an arbitrary finite degree i from the node perspective in the case where eC + h%(P\ 3 ) <C 1 

Ai = 0(l) (18) 

Y i = 0(eC + h 2 (P h )) (19) 

and the following properties hold for the degree distributions from the edge perspective: 

^ = 0\- \ ) (20) 



In 



eC+h 2 {Pb) 



(f£±Wft)\ (21) 

V ln eC+h 2 {P h ) J 
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For the case where the transmission takes place over the BEC, the bounds above are tightened by replacing h,2(P\>) 
with Pb- 

Remark 6: [On the connection between Theorems [J and |2) Theorem[2]implies that for any capacity-approaching 
LDPC ensemble whose bit error probability vanishes and also for any finite degree i in their Tanner graphs, the 
fraction of variable nodes and parity-check nodes of degree i tends to zero as the gap to capacity (e) vanishes. 
This conclusion is consistent with Theorem Q] which states that the average left and right degrees of the Tanner 
graphs scale at least like In | ; hence, these average degrees necessarily become unbounded as the gap to capacity 
vanishes. 

Corollary 3: Under the assumptions of Theorem |2j in the limit where the bit error (or erasure) probability of a 
sequence of LDPC ensembles vanishes asymptotically (as we let the block length tend to infinity) and the design 
rate is a fraction 1 — e of the channel capacity, the following properties hold for an arbitrary finite degree i 



A, 



0(1) 



o 



Pi 



0(e), 



O 



In- 



We turn now our attention to ensembles of LDPC codes which achieve vanishing bit error (or erasure) probability 
under the sum-product decoding algorithm. The following theorem relies on information-theoretic arguments and 
the stability condition, and it provides an upper bound on the fraction of degree-2 variable nodes for a sequence of 
LDPC ensembles whose transmission takes place over an arbitrary MBIOS channel. As we will see, the following 
theorem provides a tight upper bound for capacity-achieving sequences of LDPC ensembles over the BEC. 

Theorem 3: [On the fraction of degree-2 variable nodes of LDPC ensembles] Let { (n m , A(x), p(x)) } m>1 
be a sequence of ensembles of LDPC codes whose transmission takes place over an MBIOS channel. Assume that 
this sequence asymptotically achieves a fraction 1 — e of the channel capacity under iterative sum-product decoding 
with vanishing bit error probability. Based on the notation in Theorem [T] the fraction of degree-2 variable nodes 
satisfies 



e r (l - C) / eC 
A 2 < — ^ I 1 + 



1 - C 



1 + 



In ± 

91 



In 



(22) 



where 



In 



a(l)e 2 dl 



(23) 



is the Bhattacharyya distance which only depends on the channel. For sequences of LDPC ensembles whose 
transmission takes place over a BEC with an erasure probability p, if the bit erasure probability vanishes under 
iterative message-passing decoding, then the bound on the fraction of degree-2 variable nodes is tightened to 



A 2 < ~ I 1 + 



e(l-p) 
P 



1 + 



ln(l-p+§) 



(24) 



Corollary 4: Under the assumptions of Theorem [3l in the limit where the gap to capacity vanishes under the 
sum-product decoding algorithm (i.e., e — ► 0), the fraction of degree-2 variable nodes satisfies 

e r (l-C) 



Ao < 



(25) 



The tightness of the latter upper bound on the fraction of degree-2 variable nodes for capacity-achieving sequences 
of LDPC ensembles under iterative sum-product decoding is considered in Discussion [5] which follows the proof 
of Theorem [3] We state there two conditions which ensure the tightness of the bound in ( f25l >. 

Remark 7: Note that for capacity-achieving sequences of LDPC ensembles whose transmission takes place over 
the BEC, the bound in (1251 ) is particularized to \ regardless of the erasure probability of this channel. This is 
indeed the case for some capacity-achieving LDPC ensembles over the BEC (see, e.g., [12], [17], [24]). 
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Remark 8: Let us consider sequences of LDPC ensembles which achieve the capacity of a BEC under iterative 
message-passing decoding. As mentioned above, for all such known sequences, the fraction of variable nodes of 
degree 2 tends to | as the gap to capacity vanishes. This is in contrast to the behavior of the right degree-distribution 
where, for any fixed degree i, Corollary [3] implies that the fraction of the parity-check nodes of degree i tends to 
zero as the gap to capacity vanishes. 

In order to complement the picture for degree-2 variable nodes, we provide in the following theorem an upper 
bound on the fraction of edges connected to degree-2 variable nodes, and show that this fraction of edges vanishes 
as the gap to capacity under iterative decoding tends to zero. Like the previous theorem, the derivation of the 
following theorem also relies on information-theoretic arguments and the stability condition. 

Theorem 4: [On the fraction of edges connected to degree-2 variable nodes] Under the assumptions of 
Theorem the fraction of edges connected to variable nodes of degree 2 satisfies 

e r ln(^ 

\ 2 <— / ^ (26) 

ln( _ M 



[1-2^(1=^0)]' 



where g\ and r are introduced in (fl2l and (|23T ). respectively. For a BEC with erasure probability p, the bound in 
(1261 ) is tightened to 

In 



A2 < , P 1pV (27) 

pln(l-p + fj 

Corollary 5: [A looser and simple version of the bounds in Theorem 3) The upper bound on the fraction of 
edges connected to degree-2 variable nodes can be loosened to 

A 2 < — (28) 



c 1 + c 2 ln(i) 

for some constants c\ and c 2 which only depend on the MBIOS channel, and where [x] + = max(x,0); the 
coefficient c 2 of the logarithm in (|28T ) is given by 

— T 

<=2 = -4— r- < 29 > 

Hi) 

and it is strictly positive. 

Theorem 0] and Corollary [5] show that the fraction of edges connected to degree-2 variable nodes of capacity- 
achieving LDPC ensembles tends to zero, though the decay of the upper bound on A 2 to zero is rather slow as the 
gap to capacity vanishes. In the following proposition, we show that for transmission over the BEC, the bounds in 
(1171) and ((281) are indeed tight. 

Proposition 1: [On the tightness of the upper bound on A 2 for capacity-achieving sequences of LDPC 
ensembles over the BEC] The upper bounds on the fraction of edges connected to degree-2 variable nodes in (l27l ) 
and (1281 ) are tight for the sequence of capacity-achieving right-regular LDPC ensembles over the BEC in [24]. For 
this capacity-achieving sequence, A 2 = A 2 (e) decays to zero as e — > similarly to the upper bound in d28l ) with 
the same coefficient c 2 of In (-) as given in 



IV. Proofs and Discussions 

A. Proof of Theorem [7J 

Let X be a random codeword from the binary linear block code C. Let Y designate the output of the commu- 
nication channel when X is transmitted. Based on the assumption that the code C is represented by a full-rank 
parity-check matrix and that Q is the corresponding bipartite (Tanner) graph representing this code, the conditional 
entropy of X given Y satisfies the inequality in ([8]) (the derivation of this lower bound on the conditional entropy 
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is outlined in Section ITl-Ct for the full derivation, which includes all the mathematical details, the reader is referred 
to [25]). Using the convexity of f(t) = x l for all x > and applying Jensen's inequality gives 

T(x) = ^T i x i >x^ ir ' =x aR , x>0. 



Substituting the inequality above in ([8]) implies that 

#(X|Y) 



n 



9k 



21n2 ^ k(2k - 1) ' 



To continue, we apply the following lemma, which relates g\ and in © for all k G N. 
Lemma 3: 

g k > (gi) k , VfceN 

where is defined in ©. 

Proof: For k = 1, (1311 ) is trivial. For k > 1, the equality in (fTOb and Holder's inequality give 



9k =E 



tanh 2fc (- 
V2 



E* 



> E 



tanh 



L 



(tanh 2 ^) 



(51 



(30) 



(31) 



The substitution of (131b in (1301 ) gives 

H(X\Y) 



(.91 



n 21n2 ^ jfc(2ifc - 1) 

Expanding the binary entropy function into a power series around | (see [25, Appendix II.A]) gives 

^ [l-2x) 2fc 



h 2 (x) = 1 



— r- 

In 9 ^ 



2 In 2 fc(2fc - 1) 



< x < 1 



and assigning x = 1 ^ gives 



1 ^ u k 
21n2 ^ fc(2& - 1) 



,1 — \AI N , 
1 - h 2 I — 1 , <u < 1. 



(32) 



(33) 



(34) 



Since < tanh 2 (x) < 1 for all x 6 R, we get from (|9]) that < g\ < 1 (this property holds for all the sequence 
telfcli)- Substituting (O into ([32]) gives 



#(X|Y) 



>.R-C+(l-i?) 



1 - /lo 



1 u 

1 ~9i 



a R /2 N 



Fano's inequality provides the following upper bound on the conditional entropy of X given Y: 

ff(X|Y) 



n 



<Rh 2 (P h ) 



(35) 



(36) 



where P\> designates the bit error probability of the code under some arbitrary decoding algorithm (without any 
loss of generality, one can assume that the first nR bits of the code are its information bits, and their knowledge 
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is sufficient for determining the codeword). In order to make later the statement also valid for code ensembles (as 
is clarified later in Discussion [2), we loosen the bound in (l36l) . and get 



H(X.\Y) 



< h 2 (P b ) 



Combining the upper bound on the conditional entropy in (I37T ) with the lower bound in (1351 ) gives 



h 2 (P b ) >R-C+(1-R) 



1 - h 7 



1 u 

l ~9i 



(37) 



(38) 



Since the RHS of (I38T ) is monotonically increasing in R, then following our assumption, one can possibly loosen 
the bound by replacing R with (1 — e)C; this yields after some algebra that 



ho 



ILL 
~9i 



an/2" 



> 



l-C-h 2 {P h ) 
1 - (l-e)C 



Since the binary entropy function h 2 is monotonically increasing between and | then 

51 " ^ V l-(l-e)C J 

which gives the lower bound on or in (fTTT) (note that g\ < 1 and this inequality is strict unless the channel is 
noiseless). 

Let us now consider the particular case where the transmission is over the BEC. Note that for a BEC with erasure 
probability p, = 1 — p for all k G N and therefore (l32l is particularized to 



H(xm >R _ c + L i-R)(i-py 



n 



2 In 2 



2 k(2k- 1) ' 



Substituting u = 1 in (134] ) gives the equality 



1 00 1 

-J- V — 

21n2 f-J fc(2fc - 1 



and hence, the last inequality gets the form 

F(X|Y) 



> R -C + (1 -#)(!- p) c 



(39) 



Note that the RHS of (|39l is monotonic increasing with the code rate R; following our assumption that R > (1— e)C 
and since C = 1 — p is the channel capacity of the BEC, we get 

ff(X|Y) 



n 



> -e(l - p) + (1 - (1 - e)(l - p)) (1 - p) a * . 



(40) 



The normalized conditional entropy — * — ! — '- satisfies 



n 



(a) 1 

n 



i=l 



i=l 

< -£fT(^|Y) 



(6) 



i=l 

< n 



(41) 
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where (a) holds since the code dimension is at most nR and assuming without any loss of generality that the first 
nR bits form the information bits of the code, and (b) holds since given Y, the decoder finds Xj with probability 
1 — -Pb and otherwise the bit X; is erased and due to the linearity of the code it takes the values and 1 with equal 
probability so its entropy is equal to 1 bit. Combining (l40b with (|41T) gives 



Pb > ~e(l - p) + (1 - (1 - e)(l - p)) (1 - p) aa • (42) 

Finally, the lower bound on the average right degree in ( fT3l) follows from (1421) by simple algebra. Note that in the 
case where Pb = 0, the resulting lower bound coincides with the result obtained in [22, p. 1619] (though it was 
obtained there in a different way), and it gets the form 



or > 



ln ( 1 + IT^ 



In 



i 

l-p 



(43) 



We wish now to show that among all the MBIOS channels with a fixed capacity C and for fixed values of 
the gap to capacity (e) and the bit error/ erasure probability (Pb), the lower bound on the average degree of the 
parity-check nodes as given in (ITTb attains its maximal and minimal values for a BSC and BEC, respectively. To 
this end, note that for fixed values of C, e and P,, the numerator of the lower bound in (flTT) is fixed; hence, the 
value of this lower bound is maximized or minimized by maximizing or minimizing, respectively, the value of g\ 
as given in (fl2l) (note that g\ lies in general between zero and one). 

Lemma 4: [Extreme values of g\ among all MBIOS channels with a fixed capacity] Among all the MBIOS 
channels with a fixed capacity C, the value of g\ satisfies 



C < gi < (1 



2h 2 \l 



(44) 



and these upper and lower bounds on g\ are attained for a BSC and BEC, respectively. 

Proof: The proof of the lower bound on g\ follows from the calculations in [25, p. 565], though the proof 
there is not explicit (since something else was needed to prove there). We prove here this lower bound explicitly 
to improve readability. From d33l) , we get that for < x < 1 



h 2 {x) 



1 



1 oo 

— y 

n 9 ^ 



(1 - 2x) 



21, 



2 In 2 



> 1 



" k(2k 
(l-2x) 2 g 



1) 



2 In 2 



' k(2k 



1 - (1 - 2x) 



(45) 



where the last inequality holds since (1 — 2x) 2k < (1 — 2x)' 2 for k £ N and < x < 1, and the last transition 



follows from the equality X^fcLi 



l 

2fc(2fc-l) 



1 



In 2. By substituting x 



l+e 1 



for I > in (|45J), it follows that 



l + e' 



> 1 - tanri 



V/ G [0,oo). 



(46) 



The substitution of d46b in (fT2l gives 

/•OO 

gi = I a(l)(l + e~')tanh 2 
Jo 



> 



/•OO 

/ a(i)(l + e-') 

JO 



I -ho 



dl 
1 



l + e' 



dl 



C 



(47) 



where the last equality relies on a possible representation of the channel capacity for an arbitrary MBIOS channel 
(see [21, Section 4.1.8]). This completes the proof of the lower bound on g\, as given on the LHS of (l44l . Note 
that this lower bound on g\ is attained for a BEC (since for a BEC with an arbitrary erasure probability p, we get 
from C[2]> that g x = 1 - p = C). 
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In order to prove the upper bound on g\, as given on the RHS of (1441) . observe that the channel capacity of an 
arbitrary MBIOS channel satisfies 



C 



(a) 



a(l)(l + e- 1 
a{l){l + e~ l 



l-h 2 



1 



1 + e' 



dl 



y tanh 2 * (!) d> 



1 



2 In 2 



(b) 



fc=i 

— y 

In 9 ^ 



21n2 ^ A:(2/c - 1 
/ °°o(0(l + e-')tanh 



2A: 'I) dl 



k(2k- 1) 



.9A- 



(48) 



2 In 2 ^ fc(2fc - 1) 

where equality (a) follows by substituting in (133T ) x = yq^r for / > 0, and equality (b) follows from ©; this 
provides an expression for the channel capacity in terms of the non-negative sequence {gk}'j? = Q defined in (©. 
Since we look for the maximal value of gi among all MBIOS channels with a fixed capacity, then we need to 
solve the maximization problem 



maximize < g\ 



1 oo 

1 \ - 9k 

2 In 2 k(2k-l) 



C 



(49) 



Based on Lemma |3j for every MBIOS channel, g^ > {g\) k for all fceN; therefore, one can write g% = (gi) + 
where > for all k G N. By substituting this in the infinite series (|48"1) . we get 



oo 

— y 

In 9 <L*t 



9k 



21n2 ^ k(2k - 1) 



(9i) k 



-. oo 

—y 

21n2 ^ k(2k- 1) 21n2 ^ fc(2fc - 1) 



1 oo 

— F 

In 9 ^ 



l-/l 2 



E 



2 In 2 k(2k - 1) 



(50) 



where the last equality is based on (I34T >. Since > for all k G N, the equality constraint in d49l ) and the equality 
in (T50]) yield that 



1-/12 



< c 



from which the RHS of (l44l follows. Note that this upper bound on g\ is achieved when the second term in 
(l50l vanishes. Since {e/j}^ =1 is a non-negative sequence, this happens if and only if et = for all k £ N. For 
a BSC with an arbitrary crossover probability p, the LLR at the channel output is bimodal and it gets the values 
± In ( ) , which implies from (flOl that 



5fc = E 



tanh 2fe I - 



(1 - p) tanh 2fc 



+ p tanh 



2A- 



Z=-ln 



tanh 



2k 



1 



2/,- 



+ 1 
(1 - 2p) 2k 



(51) 
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Hence for the BSC, g^ = (<7i) and = for all k € N. The upper bound on g\ on the RHS of (1441 ) is therefore 
achieved for a BSC whose crossover probability is p = h^il — C). ■ 
Based on the paragraph which proceeds Lemma [4] and the result of this lemma, the last claim in Theorem Q] is 
proved. 

Discussion 1: [A discussion on the bounds in Theorem Q] and their comparison to [25, Eq. (77)]] In the 

particular case where P^ vanishes, the lower bound on the average right degree, as introduced in (fTTT ). forms a 
tightened version of the bound given in [25, Eq. (77)]. We note that both bounds are derived from the lower 
bound ([8]) on the conditional entropy of the transmitted codeword given the channel output. However, the bound 
in dTTb (even in the particular case where we set P\, to zero) is tighter than the bound given in [25, Eq. (77)]; this 
follows from the improved tightness of the lower bound on the RHS of ([8]), as obtained in the proof of Theorem Q] 
Particularly, the authors in [25] rely on the fact that all the terms of the infinite sum in the RHS of ([8]) are non- 
negative, and derive a simple lower bound on the conditional entropy by truncating this infinite sum after its first 
term. In the proof of Theorem [T] on the other hand, this lower bound on the conditional entropy is improved by 
applying Jensen's inequality and the Taylor series expansion in (l34l . Note that in the case where the gap to capacity 
vanishes (i.e., e — > 0), the average right degree tends to infinity. Since g\ < 1 where this inequality is strict unless 
the channel is noiseless, then when e becomes small, g® R << 1. As compared to the derivation of Theorem [T] 
the loss in the tightness of the bound in [25] due to the truncation of the infinite sum after its first term becomes 
marginal as the gap (in rate) to capacity vanishes; though, the difference between the two lower bounds on the 
average right degree given in © and [25, Eq. (77)]] becomes more significant as the gap to capacity is increased. 
Note also that the bound on the average right degree in Theorem Q] takes into account the bit error probability at 
the end of the decoding process, while the bound in [25, Eq. (77)] applies only to the case where P\, vanishes. The 
additional dependence of the bound (fTTb on P\, makes the bounds in Theorem Q] valid for LDPC ensembles of finite 
block length while the bound in [25, Eq. (77)] can be only applied to the asymptotic case of vanishing bit error (or 
erasure) probability by letting the block length tend to infinity. In addition to the tightening and generalization of 
the bound in [25, Eq. (77)], Theorem Q] also states the two extreme cases of (fTTT ) among all the MBIOS channels 
whose capacity is fixed. 

Discussion 2: [An adaptation of Theorem Q] for LDPC ensembles] As mentioned in Remark @] the statement 
in Theorem \T\ can be easily adapted to hold for an LDPC ensemble (n,A,/o) whose transmission takes place over 
an arbitrary MBIOS channel. This modification is done by relaxing the requirement of a full-rank parity-check 
matrix for a particular code with the requirement that the design rate of the ensemble is (at least) a fraction 1 — e 
of the channel capacity. In this case, by taking the statistical expectation over the codes from the considered LDPC 
ensemble (in addition to the original statistical expectation over the codewords of the code), the inequality in (1351 ) 
holds where we also need here to average over the rate R of the codes from the ensemble. Instead of averaging 
over the rate R of codes from the ensemble, since the RHS of (I35T ) is monotonically increasing with R, one can 
replace the rate of any code from this ensemble with the design rate of the ensemble (which forms a lower bound 
on R for every code from this ensemble). By assumption, the design rate is not less than (1 — e)C\ hence, (1351 ) 
and the requirement on the design rate yield that 



E 



H(X|Y) 



> 1 - C- (1 - (l-e)C) E 



ho 



a R /r 
yi 



(52) 



where E designates the statistical expectation over all the codes from the LDPC ensemble (n, A,/)). 
Since the binary entropy function is a concave function, Jensen's inequality gives 



E 



h 2 



1 



1 -E 



< h? 



' a R /2" 

yi 



(53) 



and another application of Jensen's inequality to the exponential function (which is convex) gives 

E 



a R /2 

Si 



EM/ 2 
— 9l 



(54) 



Note that < g\ < 1 and for every code from the ensemble or > 2 (since pi = for i = 0, 1, i.e., the degrees of 
all the parity-check nodes are at least 2, then also the average right degree of any code from the ensemble cannot 
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be smaller than 2); this implies that the arguments inside the binary entropy functions of d53l ) lie between zero and 
one-half. 

The loosening of the bound in the transition from 061 ) to 071 ) is due to the fact that we need an upper bound 
on the rate R of a code from this ensemble; since we consider binary codes, a trivial upper bound on the rate is 1 
bit per channel use (note that the rate of an arbitrarily chosen code from this ensemble may exceed the channel 
capacity). Note also that due to the concavity of the binary entropy function and Jensen's inequality, one can replace 
Pb on the RHS of 071 ) which applies to a particular code with the average bit error probability of the ensemble, 
and the upper bound on the conditional entropy still holds. This gives 

"FfXIYl 



hnCK) >E 



71 



(55) 



where Pb = E [Pb] designates the average bit error probability of the LDPC ensemble. 

The combination of the chain of inequalities in (I52l)-d55l) leads to the adaptation of the statement in Theorem Q] 
for arbitrary LDPC ensembles with the proper modification of the requirement on the design rate of the ensemble 
(instead of the requirement of a full-rank parity-check matrix which is quite heavy for individual codes which are 
selected at random from an LDPC ensemble), and the reference to the average bit error (or erasure) probability of 
the ensemble. 

Note that the adaptation of the statement in Theorem Q] for LDPC ensembles whose transmission takes place 
over the BEC is more direct. In this case, we refer to ((42]) : since in the latter case, /i2(Pt>) is replaced by Pb, then 
two uses (out of three) of Jensen's inequality become irrelevant for the BEC. 

Discussion 3: [An adaptation of Theorem Q] for LDPC ensembles with random or intentional puncturing 
patterns] In continuation to Discussion it is also possible to adapt Theorem Q] to hold for ensembles of punctured 
LDPC codes whose transmission takes place over an MBIOS channel. To this end, we refer the reader to [23, 
Section 5] which is focused on the derivation of lower bounds on the average right degree and the graphical 
complexity of such ensembles. The derivation of these bounds relies on a lower bound on the conditional entropy 
of the transmitted codeword given the received sequence at the output of a set of parallel MBIOS channels (see 
[23, Eqs. (2) and (3)]); the latter bound was particularized in [23, Sections 2-4] to the two settings of randomly 
and intentionally punctured LDPC ensembles which are communicated over a single MBIOS channel. We note 
that the lower bound on the conditional entropy, as given in ([8]), forms a particular case of the bound in [23, 
Eqs. (2) and (3)] where the set of parallel MBIOS channels degenerates to a single MBIOS channel. The concept 
of the proof of Theorem [T] enables one to tighten the lower bounds on the average right degree and the graphical 
complexity, as presented in [23, Section 5], for both randomly and intentionally punctured LDPC ensembles. More 
explicitly, by comparing the proof of (fTTI) with the derivation of [25, Eq. (77)] under the assumption of vanishing 
bit error probability, one notices that the tightening of the bound in the former case follows by combining Lemma [3] 
with the equality in 04l (instead of the truncation of a non-negative infinite series after its first term, as was done 
for the derivation of the looser bound in [25]); this difference can be exploited exactly in the same way in [23, 
Section 5] for improving the tightness of the lower bounds on the average right degree and the graphical complexity 
for punctured LDPC ensembles. 

Discussion 4: [Lower bounds on the conditional entropy which are based on statistical physics] Theorem [T] 
is proved via the lower bound on the conditional entropy ([8]) which holds for an arbitrary binary linear block 
code that is represented by a full-rank parity-check matrix and whose transmission takes place over an MBIOS 
channel. This bound is adapted above to hold for LDPC ensembles under ML decoding or any sub-optimal decoding 
algorithm (see Discussion [2]). A different approach for obtaining a lower bound on the same conditional entropy 
for ensembles of LDPC codes and ensembles of low-density generator-matrix (LDGM) codes was introduced by 
Montanari in [15]. This bounding technique is based on tools borrowed from statistical physics, and it provides 
lower bounds on the entropy of the transmitted message conditioned on the received sequence at the output of 
the MBIOS channel. The computational complexity of the bound in [15] grows exponentially with the maximal 
right and left degrees (see [15, Eqs. (6.2) and (6.3)]) which therefore imposes a difficulty on the calculation of this 
bound (especially, for continuous-output channels). Since the bounds in [15] are derived for ensembles of codes, 
they are probabilistic in their nature; based on concentration arguments, they hold asymptotically in probability 1 as 
the block length tends to infinity. Based on heuristic statistical mechanics calculations, it was conjectured that the 
bounds in [15], which hold for general LDPC and LDGM ensembles over MBIOS channels, are tight. As opposed 
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to the lower bound on the conditional entropy which is given in ([8]), the bounding techniques in [14] and [15], 
where both rely on statistical physics, do not provide a bound which is valid for every binary linear block code. 
It would be interesting to get some theory that unifies the information-theoretic and statistical physics approaches 
and provides bounds that are tight on the average and valid code by code. 

Proof of Corollary [JJ 

From Remark |2j the cardinality of the fundamental system of cycles of the Tanner graph Q, which is associated 
with a full spanning forest of Q, is equal to the cycle rank (3(G)- From Eq. ([2]), we get that (3(G) > \Eg\ — \Vg\ where 
\Eg\ and \Vg\ designate the number of edges and vertices. Specializing this for a Tanner graph G which represents 
a full-rank parity-check matrix of a binary linear block code, the number of vertices satisfies \Vg\ = n(2 — R) 
(since there are n variable nodes and n(l — R) parity-check nodes in the graph) and the number of edges satisfies 
\Eg\ = n(l — i2)aR. Combining these equalities gives the lower bound on the cardinality of the fundamental system 
of cycles in I®. 

Proof of Corollary [2] 

The proof of ( fT6l ) and ( fTTl ) is based on Remark 0] and Corollary [JJ From the substitution of Pi, = in (fTTb . 
which is justified in Discussion [2] for LDPC ensembles, one obtains the following lower bound on the average right 
degree of the LDPC ensemble as the average bit error probability of this ensemble vanishes: 




l-2h: 



Hi 



(56) 



Since we assume here that the bit error probability of the ensemble vanishes as the block length tends to infinity, 
then asymptotically with probability 1, the code rate of an arbitrary code from the considered ensemble does not 
exceed the channel capacity. By substituting the lower bound on a R from (l56l) and an upper bound on R (i.e., 
R < C) into (ITST t. the asymptotic result in (fl6l ) readily follows. The proof of (fTTT ) follows similarly based on ([131) 
(with P, = 0). From the last statement in Theorem [TJ regarding the maximal and minimal values of the lower bound 
in (fTTl ) among all the MBIOS channels with a fixed capacity, it follows that the same conclusion also holds w.r.t. 
the lower bound in ( fT6t . Hence, the RHS of (IT6T ) also attains, respectively, its maximal and minimal values for the 
BSC and BEC whose capacities are equal to C. 



B. Proof of Theorem [2] 



Since the fraction of variable nodes of degree i is not greater than 1 for any degree i, (fl8[ ) clearly holds (we 
demonstrate later that this result is asymptotically tight as the gap to capacity vanishes, at least for degree-2 variable 
nodes). 

We turn now to consider the degrees of the parity-check nodes. Similarly to the proof of Theorem [TJ we denote 
by X a random codeword from the ensemble (n, A, p) where the randomness is over the selected code from the 
ensemble and the codeword which is selected from the code. Let Y designate the output of the communication 
channel when X is transmitted. The lower bound on the conditional entropy of X given Y in [25, Eq. (56)] gives 



H(X\Y) 



n 



>R-C + 



1 - R 

2 In 2 



E 



-eC + 



- k ( 2k 
l-(l-e)C g 

i=2 



2 In 2 



9k 



' k(2k - 1) 



(57) 



where the equality follows from the definition of T and since the design rate of the ensemble satisfies R = (1 — e)C. 
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Applying Lemma [3] to the RHS of d57l ), we get 

H(X\Y) 



n 



(9\ 



2 In 2 



i=2 
oo 



i=2 



k=l 
1-/12 



k(2k- 1) 



i/2 N 



where the last equality follows from ( 1341) . Combining the upper bound on the conditional entropy in (1361 ) with the 
last result gives 



h 2 (P h )>-eC+(l-(l-e)C)J2 



i=2 



1 - ho 



i/2- 



and therefore 



£ 

j=2 



i/2 s 



< 



eC + h 2 (P b ) 
1 - (l-e)C 



(58) 



where Pb designates the average bit error probability of the ensemble under the considered decoding algorithm. 
Since all the terms in the sum on the LHS of d58l ) are non-negative, this sum is lower bounded by its i'th term, 
for any degree i. This provides the following lower bound on the fraction of parity-check nodes of degree i: 

eC + h 2 {P b ) 1 



< {eC + h 2 (P h )) 



l-C 



(59) 



This completes the proof of the statement in ( fT9l when the transmission takes place over an arbitrary MBIOS 
channel. Let us now consider the particular case where the transmission is over a BEC with erasure probability 
p. In this case, = 1 — p for all i e N, and the channel capacity is given by C = 1 — p. Therefore, (1571 ) is 
particularized to 



2 In 2 



i=2 
oo 



oo _. 



-e(l - p) + (1 - (1 - e)(l - p)) £ I\ (1 - p)* 



(60) 



i=2 



where the equality holds since Y1T=1 fc(2fc-i) 
to the LHS of ([6(5]), we get 



2 In 2. Applying the upper bound on the conditional entropy (|4TT) 



Pb > ~e(l - p) + (1 - (1 - e)(l - p)) T i C 1 " P)' 



i=2 



and therefore 



E{a-p)T,} £ r^^, 



i=2 



(61) 



where this time Pb denotes the average bit erasure probability of the ensemble. Following the same steps as above, 
we get that for the BEC 

e(l-p)+P h 



v i_(i_ e )(i_p))(i_p)< 



(62) 
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so indeed h^Pb) is replaced with P D in CDS'- 

We turn now to consider the pair of degree distributions from the edge perspective. The average left degree (ol) 
of an LDPC ensemble satisfies 



1 

ol Jo 



i 

\(x) dx 

,\ oo 



r 1 ^ 

/ Vv^fc 



i=2 

oo 



= E- < 63 ) 
U 1 

which implies that the fraction of edges connected to variable nodes of an arbitrary degree i satisfies 

Aj < — . (64) 

Since the design rate of the LDPC ensemble is assumed to be a fraction 1 — e of the channel capacity, then the 
average right and left degrees are related via the equality 

a L = (l-(l-e)C)a R . (65) 

Substituting (|65T ) on the RHS of (l64l and applying the lower bound on or in (fTTb gives 

A* < ^ . (66) 

2(l-(l-e)C) In 

'2 I l_(i_ e )0 

Using the power series for the binary entropy function in (1331) and truncating the sum on the RHS after the first 
term gives 

, / s (l-2x) 2 

l-h 2 (x) > - '—. 

w ~ 2 In 2 

Substituting u = li2(x) yields 

(l - 2h' 2 1 {u)) 2 < 2 ln2 • (1 - u) . (67) 
Substituting (f6Tb into the denominator on the RHS of d66l ). we get 



A,; < 



V l-(l-e)C / 



iln(i) 



zln(i) 



i \ , i„ n-c^ 

.2 In 2/ 



which completes the proof of (I20t for general MBIOS channels. For the BEC, we substitute (1651) and the lower 
bound on the average right degree in (fT3l into the RHS of (l64l) to get 

A, < {l ~ p) 



(l-(l- e )(l-p)) ln l + ^g 



£ (l-p) + Pb 



(l-(l- £ )(l-p)) In(^^ 
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Hence, /j2(Pt>) is replaced by P, in (l20l ) when the communication channel is a BEC. Considering the right degree 
distribution of the ensemble, we have 

1 f 1 

— = / p(x) dx 

«R Jo 

oo 

i=2 

Replacing (l63l ) with the above equality and following the same steps as above, one obtains an upper bound on the 
fraction of edges connected to parity-check nodes of a given degree i. The asymptotic behavior of the resulting 
upper bound on pi is similar to the upper bound on A» as given in d68l ). However, a tighter upper bound on the 
fraction of edges connected to parity-check nodes of degree i is derived from the equality 

Pi = —. (69) 
Substituting (fTTT ) and d59l in the above equality, we get 

ftS ££±M» K*, — — m 

2 In 



-, n h -l I l-C-h 2 (P b ) 

1 Z ""> 1 l_(l-e)C 



Applying (I67T ) to the denominator of the second term on the RHS of (f70T > gives 

„ ec + /i 2 (p b ) ln (ir 

Pi — 



l-(l-e)C 



eC + h 2 (P b ) ln (^ 



1-C 1 l-(l-e)C \ 1 , 

in I 21n2 eC+Mfl.) J 1 ~ " 2 I 



< 



ln (jr) e c + /» 2 (flO 



'- C m( R ^ y ) + ln(i^) 



2 



This proves the statement in (f2TT ) regarding the fraction of edges connected to parity-check nodes of an arbitrary 
finite degree i. 

When the communication takes place over the BEC, we substitute ( fl3l ) and d62b in d69l to get 



i[e(l-p) + P b ] ln (r^ 

Pi < 



Applying some simple algebra, similarly to the previous derivations, the statement in (|2Tb is validated even when 
^2(Pb) is replaced with P,. 

C. Proof of Theorem \3\ 

The average degrees of the variable nodes and the parity-check nodes of a bipartite graph which represents a code 
from an LDPC ensemble are expressible in terms of the pair of the degree distributions (A, p) of this ensemble, 
and these average degrees are given, respectively, by a L = (Jq 1 \(x)dx) 1 and a R = (Jq 1 p(x)dx) 1 . Hence, the 
fraction of degree-2 variable nodes is given by 

A 2 = ^= / 2 (VI) 
Jo 
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and the design rate of this ensemble is given by 

R , a L Jo p{x)dx 

-Kd = J- = 1 1 • 

OR J X(x)dx 

Using the last equation, we rewrite the denominator of (1711 ) as 

i 1 ,1 
X(x)dx = — / p(x)dx . (72) 

io 

According to our assumption, the considered sequence of ensembles achieves vanishing bit error probability under 
sum-product decoding and hence the stability condition implies that 

r 

e 



where r is introduced in (|23T ). Substituting (l72l) in (TTTT ) and applying the inequality above leads to an upper bound 
on A2 of the form 

A 2 < er "7^» . (73) 

Relying on the convexity of the function f(t) = x t for all x > 0, Jensen's inequality gives 

•1 

p{x)dx 

[ Vft^Mi 

Jo i 
1 

x p '^dx 

1 















which implies that 



P(l)> rl ^— -l = a R -l. (74) 

Jo P(^)dx 

Substituting d74l in (1731 ) and since the ensembles are assumed to achieve a fraction 1 — e of the channel capacity 
(i.e., R& = (1 — e)C) under sum-product decoding with vanishing bit error probability then 

e r (l-i? d ) / 1 
£ : — ( I + 1 



e r (l - 


i2d) 


2 




e r (l - 


Rd) 


2 




e r (l - 


C) 



OR - 1 



Since the RHS of (l75l) is monotonically decreasing with average right degree, the bound still holds when <2r is 
replaced by a lower bound. For all m G N, let Pb,m designate the average bit error probability of the ensemble 
(n m , A(x), under the considered decoding algorithm. Applying Theorem[T]and letting Pb,m tend to zero gives 

2 In 



K 9i 

The upper bound in (l22l follows by substituting d76l) in d75l ). 



or > 7 — ^ • (76) 

ln( i 
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Turning to consider transmission over the BEC, the improved upper bound on the degree-2 variable nodes follows 
by substituting the lower bound in (fT3T ) (when we let the bit erasure probability to vanish) into (l75l) . Note that for 
a BEC with erasure probability p, C = 1 — p and e r (l — C) = 1. 

Discussion 5: [On the tightness of the upper bound d25l ) on the fraction of degree-2 variable nodes for 
capacity-achieving LDPC ensembles over MBIOS channels] As a direct consequence of Theorem HI then in the 
limit were the gap to capacity vanishes under the sum-product decoding algorithm, the fraction of degree-2 variable 
nodes of the LDPC ensembles satisfies the upper bound in d25l ). In the following, we consider the tightness of this 
upper bound. To this end, we first present the following lemma: 

Lemma 5: [On the asymptotic fraction of degree 2 variable nodes for capacity-achieving sequences of 
LDPC ensembles] Let 

Un m ,X m (x) A^^-I ipm{x) A 

<- ' — ' — > m£N 

i i 

be a sequence of LDPC ensembles whose transmission takes place over an MBIOS channel, and let C designate the 
channel capacity (in units of bits per channel use). Assume that this sequence approaches the channel capacity under 
the sum-product iterative decoding algorithm, and also assume that the flatness condition is asymptotically satisfied 
for this sequence (i.e., let the stability condition be asymptotically satisfied with equality for this capacity-achieving 
sequence). Let Ag be the fraction of the variable nodes of degree 2 of the m'th ensemble in this sequence. Also, 
let be a random variable distributed according to the degree distribution of the parity-check nodes of the m'th 
ensemble, and let = E [d^] designate the average right degree of this ensemble. Finally, let us assume that 
the limit of the ratio between the standard deviation and the expectation of the right degree distribution is finite, 

s td(4 m) ) 

lim = K <oo (77) 

m— >oo n^ m > 

"r 

where std(c4"^) denotes the standard deviation of the random variable . Then, in the limit where m tends to 
infinity, the fraction of degree-2 variable nodes satisfies 

lim A™ = e \ {l ~ C l (78) 

m^oo 2 2(1 +K 2 ) V ' 

where r is the Bhattacharyya distance, introduced in (|23l ), which only depends on the communication channel. 

Proof: By the assumption that the sequence of LDPC ensembles satisfies asymptotically the flatness condition, 
we get 

lim 4 m) p' m (l) = e r . (79) 



- oc 



Based on the equality in ( TTTb 



, (m) 

A (m) = __A2 ; \/m G N (80) 

A m (x)dx 



o 



and therefore 



(m) 

lim 



lim A 2 

m— *oo 
(a) 



1 

2p' m (l) I X m (x)dx 





(b) Hm e^(l-R 



l 

2 Pm( 1 ) I Pm{x)dx 











lim (81) 

P'miX) / Pm(x)dx 
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where the equality in (a) relies on d80b and the requirement in d79l ), the equality in (b) follows from £[)) where 
R m designates the design rate of the ensemble (n m , A m , p m ), and (c) follows by the assumption that the sequence 
of LDPC ensembles is capacity-achieving. Based on the convexity of the function f(t) = x l for all x > 0, then 
Jensen's inequality implies that for all m € N and non-negative values of x 

Pm{x) = Y.P { r ] * l ~ l > xZ*<*-Vf>t> = x P' m (D . 
i 

Integrating both sides of the inequality over the interval [0, 1] gives 

f p m (x)dx > [ Xf'^dx ' 

Jo Jo 

which therefore implies that 



(L0) + 1 



p m (l)>^T^ l = a { ™ ) -l. (82) 

p m (x)dx 



o 

From Theorem [T] the asymptotic average right degree of a capacity-achieving sequence of LDPC ensembles tends 
to infinity as the gap to capacity vanishes. Therefore, (l82l) implies that lim m ^oo p' m (l) = oo . Substituting this into 
the RHS of dHB gives the equality 

lim = lim L_ . (83 ) 

m— >oo z m—>oo / x 

(p'mi 1 ) + 1) / Pm(x)dx 
JO 

Let r| m ^ be the fraction of parity-check nodes of degree i in the m'th LDPC ensemble, then the following equality 
holds: , 

■ -n( m ) 

Pi - 



r 



(m) 



and 



P'raiX) + 1 

= E{(*- 1 >S m) } + 1 

i 

E. (m) 

_ S>«rj m) 

The denominator in the RHS of (f83T > therefore satisfies the following chain of equalities: 

(p'mi 1 ) + 1) / Pm( a; ) (ix 
JO 



(84) 



E,-i r 



2 r M 



E 7 ir 



(m) 
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where var(d^ ) designates the variance of the random variable dp, ■ Taking the limit where m tends to infinity 
on both sides of (l85l) . the definition of K in (1771 ) yields that 

lim (p' m (l) + 1) / p m (x)dx =1+K 2 . 

The proof of this lemma is completed by substituting the last equality in the RHS of (f83T >. ■ 
Lemma [5] shows that for capacity-achieving sequences of LDPC ensembles for which the following assumptions 
hold: 

• Condition 1: The stability condition is asymptotically satisfied with equality as the gap to capacity vanishes 
(i.e., the flatness condition is asymptotically satisfied) 

• Condition 2: The limit of the ratio between the standard deviation and the expectation of the right degree 
distribution is finite (see (1771 )) 

then the asymptotic fraction of degree 2 variable nodes is bounded away from zero as the gap to capacity vanishes 
(under the sum-product decoding algorithm). Moreover, if the limit in Condition 2 is equal to zero (i.e., K = 0), 
then the asymptotic fraction of degree-2 variable nodes coincides with the upper bound in (12"5l ): this makes the 
upper bound in (1231 ) tight under the above two conditions for capacity-achieving sequences of LDPC ensembles 
under the sum-product decoding algorithm. 

As an example of a capacity-achieving sequence of LDPC ensembles which satisfies the upper bound in (I23T ) 
with equality, consider the sequence of right-regular LDPC ensembles as introduced by Shokrollahi (see [17], [24]); 
this capacity-achieving sequence satisfies the flatness condition on the BEC; also, by definition, K = for this 
sequence since the degree of the parity-check nodes is fixed for a right-regular LDPC ensemble. Note that for the 
BEC, the upper bound on the fraction of degree-2 variable nodes as given in (1231 ) is particularized to one-half 
regardless of the erasure probability of the BEC. 

Remark 9: We note that the property proved in Lemma [5] for the non-vanishing asymptotic fraction of degree-2 
variable nodes of capacity-achieving sequences of LDPC ensembles is reminiscent of another information-theoretic 
property which was proved by Shokrollahi with respect to the non-vanishing fraction of degree-2 output nodes for 
capacity-achieving sequences of Raptor codes whose transmission takes place over an MBIOS channel (see [5, 
Theorem 11 and Proposition 12]). The concepts of the proofs of these two results are completely different, but 
there is a duality the two results. 



D. Proof of Theorem [?] and its Corollary 

Since the considered sequence of ensembles achieves vanishing bit error probability under sum-product decoding, 
the stability condition implies that 

Aa = A ' (0) < Tii) (86) 

where r is given by (l23l) . Substituting (1741) in (l86l ) yields 

T 

A 2 < — ^-r (87) 
Or - 1 

where or designates the common average right degree of the sequence of ensembles. The upper bounds on A2 in 
(|26l ) and (1271 ) are obtained by substituting (1761 ) and d43l ), respectively, in (1871 ). 

Discussion 6: [Comparison between the two upper bounds on the fraction of edges connected to degree-2 
variable nodes: ML versus iterative decoding] In the proof of Theorem |2j we derive an upper bound on the 
fraction of edges connected to variable nodes of degree i for ensembles of LDPC codes which achieve a bit error 
(or erasure) probability P], under an arbitrary decoding algorithm (see (l66l ) and the tightened version (l68l ) of this 



26 



SUBMITTED TO IEEE TRANSACTIONS ON INFORMATION THEORY, SEPTEMBER 2007 



bound for the BEC). Referring to degree-2 variable nodes and letting P\, vanish, this bound is particularized to 

\g 1 J 



< 



< 



;i-(l-e)C) In I 



(88) 



In ( 

with the following tightened version for the BEC: 

A 2 < 



(l-(l-e)(l-p))ln 1 



e(l-p) 



< 



P 



ln(l-p+f)+ln (jh- p 



(89) 



It is interesting to note on some similarity between the upper bounds on A2 as given in (1881 ) and (1891 and the 
corresponding bounds given in (l26l ) and (|27T ). Note that the bounds in (I88T ) and (|89l are valid under ML decoding 
or any other decoding algorithm while the two bounds in (l26l ) and (|2"7T ) are more restrictive in the sense that they are 
valid under the sum-product decoding algorithm. These two pairs of bounds are numerically compared in Section |\H 
Proof of Corollary^ A truncation of the power series on the LHS of (l34l) after the first term gives the inequality 



1 - ho 



l-y/u 



> 



11 



2 In 2 



Assigning u = (l — 2h 2 1 (a^)) 2 and rearranging terms gives 



Assigning < x 



l-C 
l-(l-e)C 



KHx) > -(l- ^21n2 (1 



1 - C 



< u < 1. 



< x < 1. 



(90) 



< 1 in ([901) gives 



7^ 



> 



and 



1 - 2h 



> 



-1 



1 - (7(1 -e) 



'2 In 2 



1 - (l-e)C 



1 



'2 In 2 



e(7 
l-C 



1 - (7 



1 - (l-e)C 



< 



'21n2 



g(7 
1 - C 



Substituting (I9T1 ) in (1761) provides the following lower bound on the average right degree of the ensembles: 



OR > 



In 



1 l-C 
2 In 2 eC 



ln(^ 

\9i 



(91) 



(92) 



As, clearly, the average right degree of an LDPC ensemble is always greater than 1, then it follows from d92l ) that 

OR - 1 > 



111 I 2 ln2 eC y 



ln(i 

\9i 
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The proof is completed by substituting d93l ) in d87l ). 



E. Proof of Proposition [7J 



When the transmission takes place over a BEC whose erasure probability is p, the constant c 2 given in d29| ) takes 
the form 

c 2 = /, \ • (94) 



In 



For < a < 1, let 



A a (x) = !-(!- x) Q = ^(-l) fc+1 



fc=i 



< x < 1 



p a (x) = x°. 



(95) 



Note that all the coefficients in the power series expansion of A Q are positive for all < a < 1. Let us now define 
the polynomials X a: N and X 0: n, where X a ,N is the truncated power series of X a , where only the first N — 1 terms 
are taken (i.e., X 0: n is of degree N — 1), and the polynomial 



K,n(x) 
A q ,at(1) 



(96) 



is normalized so that A Qj tv(1) = 1. The right-regular sequences of LDPC ensembles in [24] are of the form 
{ (n m , A Qj tv(x), p a {x)) } m> i where < a < 1 and N G N are arbitrary parameters which need to be selected 
properly. Based on the analysis in [22, Theorem 2.3], this sequence achieves a fraction 1 — e of the capacity of the 
BEC with vanishing bit erasure probability under message -passing decoding when a and ./V satisfy 



and 



where 



N = max 



— — = 1 — p 



1 - k 2 (p) (1 - p) (1 - e) 



(1-pY 



k 2 {p) = (1 -p) e e 



(97) 

(98) 
(99) 



and 7 is Euler's constant. Combining (1951 ) and (1961 ). and substituting 



N-l , 

E(-D fc+1 U 
k=i v 



gives 



N-l, 



A a ,Ar(x) 



-( a )(-l) N+1 



-D k+1 (")x k 



1 " f (-1)™ &) 

Therefore, the fraction of edges adjacent to variable nodes of degree two is given by 

a 



A 2 



1 - f (-1)^+1 (-) • 
We now obtain upper and lower bounds on A2. From [22, Eq. (67)] we have that 



where 



c(Q,AQ N N +1 fa\ J_ 
N a a { ' \NJ ~ N a 

c(a, N) = (1 - a) T e !^-^^) . 



(100) 

(101) 
(102) 
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Substituting dlOlb in dlOOb and using (|97]>, we get 



a 



< A 2 < 



o 



l-c(a,N)(l-p) ^"'-l-(l-p) p 
Under the parameter assignments in d971 ) and d98l ), the parameters N and a satisfy 



a 



N > 



1 l— p 



IniV 
l-(l-p) fc 2 (p) 

e 



(103) 



(104) 



(105) 



Substituting (11041 ) and dl05l ) into the right inequality of d 1031 > gives an upper bound on A 2 which takes the form 

a 



A 2 < - 
P 



< 



p In 



l-(i-p) fc 2 (p) 
e 

V 1-P 



P 



lni+ln(l-(l-p) fc 2 (p)) 



c 3 + c 2 In i 

where c 2 is the coefficient of the logarithmic growth rate in i, which coincides here with (l29l ). and 

p ln(l - (1 -p) k 2 (p)) 



C3 



III [A- 

1— p 



(106) 



(107) 



is a constant which only depends on the BEC. We turn now to derive a lower bound on A 2 for the asymptotic 
case where the gap to capacity vanishes. From (|98T ). we have that for small enough values of e, the parameter N 
satisfies 

■\-k 2 {p)(l-p){l-ey 



N 



< l-k 2 (p)(l-p)(l-e) | 1 



l-k 2 (p)(l-p) 1-e 1 



l 

fc 2 (p)(i- P ) 



e 

Substituting dl04b and d 108b into the left inequality of dl03l ), we get 

a p 



(108) 



A 2 > - 



> 



pi- c(a,N) (1 -p) 



e 

P 



p In 



l-c(a,JV)(l-p) 



p [In (i) + ln(l - (1 - p) k 2 (p)) + ln(l - e)] 
P 



l-(l-p)c(a,JV) 
1 



P 



c 3 + c 2 ln(i) +e(e,p) 1 - (1 - p) c(a, N) 



(109) 
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where C2 is the coefficient of the logarithm in the denominator of d28l ) and it coincides with d94"l) , C3 is given in 
rfTUTb . and 

e(e, P ) = P -^$ (110) 
In 

which therefore implies that for < p < 1 



1 



lime(e,p)=0. (Ill) 

Using the lower bound on the parameter N in (1 1 05b . then in the limit where e vanishes, N tends to infinity (this 
holds since 1 — (1 —p)k2(p) > for all < p < 1 where k 2 in introduced in (l99l)). Also, from (I98T ) and (11041 ). we 
get 

lim a = 

which, from (11021 ). yields that 

lim c(a, N) = 1 . (112) 

Substituting (|llll) and (|1 121) in (11091 ) yields that in the limit where the gap to capacity vanishes (i.e., e — > 0), the 
upper and lower bounds on A2 in (11061 ) and (11091 ) coincide. Specifically, we have shown that 

lim X 2 (e) • c 2 In ( - ] = 1 . 

Therefore, as e — > 0, the upper bound on A2 = A2(e) in Corollary [5] becomes tight for the sequence of right-regular 
LDPC ensembles in [24] with the parameters chosen in Wf\ and (|98l ). We note that the setting of the parameters 
N and a in d97]) and d98]) is identical to [22, p. 1615]. 

V. Numerical Results 

In this section, we consider sequences of LDPC ensembles which achieve vanishing bit error probability and 
closely approach the channel capacity limit under sum-product decoding. As representatives of MBIOS channels, 
the considered communication channels are the binary erasure channel (BEC), binary symmetric channel (BSC) 
and the binary-input AWGN channel (BIAWGNC). 

Example 2: [BEC] Consider a sequence of LDPC ensembles (n, A, p) where the block length (n) tends to infinity 
and the pair of degree distributions is given by 

\{x) = 0.409x + 0.202x 2 + 0.0768x 3 + 0.1971x 6 + 0.1151x 7 

p{x) = x 5 . 

The design rate of this ensemble is R = 0.5004, and the threshold under iterative message -passing decoding is 
equal to 

p n = inf —. - r ^ = 0.4810 

se(0,i] A(l - p(l - X)) 

so the corresponding channel capacity of the BEC is C = 1— p lT = 0.5190 bits per channel use, and the multiplicative 
gap to capacity is e = 1 — ^ = 0.0358. The lower bound on the average right degree in (fl~3T ) with vanishing bit 
erasure probability (i.e., P h = 0) gives that the average right degree should be at least 5.0189, and practically, since 
we consider here LDPC ensembles with fixed right degree then the right degree cannot be below 6. Hence, the 
lower bound is attained in this case with equality. An upper bound on the fraction of edges which are connected 
to degree-2 variable nodes (A2) is readily calculated based on (I87T ) with e r = (p IT ) _1 = 2.0790 and the above 
lower bound on a R (for LDPC ensembles of a fixed right degree) which is equal to 6; this gives A2 < 0.4158 as 
compared to the exact value which is equal to 0.409. The exact value of the fraction of degree-2 variable nodes is 

A 2 = ^ = A2(1 "^ )QR = 0.6130 
2 2 2 



as compared to the upper bound in (1751) . combined with the tight lower bound or > 6, which gives A2 < 0.6232. 

Example 3: [BIAWGNC] Table U considers two sequences of LDPC ensembles of design rate \ which are taken 
from [4, Table II]. The pair of degree distributions of the ensembles in each sequence is fixed and the block length 
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of these ensembles tends to infinity. The LDPC ensembles in each sequence are specified by the following pairs 
of degree distributions: 

Ensemble 1: 



A(x) = 0.170031x + 0.160460x 2 + 0.112837x 5 

+0.047489x 6 + 0.011481a; 9 + 0.091537x lc 
+0.152978x 25 + 0.036131x 26 + 0.217056x : 

Hy ' 16 16 

Ensemble 2: 



A(x) = 0.153425x + 0.147526a; 2 + 0.041539x 5 + 
0.147551a; 6 + 0.047938x 17 + 0.119555x 18 
+0.036379x 54 + 0.126714x 55 + 0.179373x 

p(x) = x 11 



The asymptotic thresholds of the considered LDPC ensembles under iterative sum-product decoding are calculated 
with the density evolution technique when the transmission is assumed to take place over the BIAWGNC; these 
calculations provide the indicated gaps to capacity as given in Table U The value of A2 for each sequence of LDPC 
ensembles (where we let the block length tend to infinity) is compared with the upper bound given in Theorem [4] 
which holds for any sequence of LDPC ensembles whose bit error probability vanishes under sum-product decoding 
with a certain gap (in rate) to capacity. Note that for calculating the bound in Theorem [4] the parameter r introduced 
in (|23l ) gets the form r = for the BIAWGNC where designates the energy per information bit over the 
one-sided noise spectral density, and we substitute here the threshold value of ^ under the sum-product decoding. 
The average right degree of each sequence is also compared with the lower bound in Theorem [TJ These comparisons 
exemplify that for the examined LDPC ensembles, both of the theoretical bounds are informative. It is noted that 



LDPC 


Gap to 




Lower bound 




Upper bound 


ense- 


capacity 


a R 


on Or 


A 2 


on A2 


mble 


(e) 




(Theorem [TJ 




(Theorem |4j 


1 


3.72 ■ 10~ a 


10.938 


9.249 


0.170 


0.205 


2 


2.22 ■ 10- a 


12.000 


10.134 


0.153 


0.185 



TABLE I 

Comparison of theoretical bounds and actual values of A 2 and or for two sequences of LDPC ensembles of design 

RATE I TRANSMITTED OVER THE BIAWGNC. THE SEQUENCES ARE TAKEN FROM [4, TABLE II] AND ACHIEVE VANISHING BIT ERROR 
PROBABILITY UNDER SUM-PRODUCT DECODING WITH THE INDICATED GAPS TO CAPACITY. 



the degree distributions of the examined LDPC ensembles were obtained in [27] by numerical search, based on the 
density evolution technique, where the goal was to minimize the gap to capacity without taking into consideration 
the values of A2 or cir. Therefore, it might be possible to construct LDPC ensembles which achieve the same gap 
to capacity with values of A2 and or which are even closer to the theoretical bounds. 

Example 4: [BSC] Table Ull considers two sequences of LDPC ensembles, taken from [27], where the pair of 
degree distributions of the ensembles in each sequence is fixed and the block length of these ensembles tends to 
infinity. The LDPC ensembles in each sequence are specified by the following pairs of degree distributions and 
design rates: 
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Ensemble 1: 

A(x) = 0.291157x + 0.189174x 2 + 0.0408389x 4 

+0.0873393x 5 + 0.00742718x 6 + 0.112581x 7 

+0.0925954x 15 + 0.0186572x 20 + 0.124064x 32 

+0.016002x 39 + 0.0201644x 44 
p(x) = 0.8x 4 + 0.2x 5 
R = 0.250 

Ensemble 2: 

A(x) = 0.160424x + 0.160541x 2 + 0.0610339x 5 

+0.153434x 6 + 0.0369041x 12 + 0.020068x 15 

+0.0054856x 16 + 0.128127x 19 + 0.0233812x 24 

+0.05285542x 34 + 0.0574104x 67 + 0.0898442x 68 

+0.0504923x 85 
p{x) = x 10 
R = 0.500 

The thresholds of the considered LDPC ensembles under iterative sum-product decoding are calculated with the 
density evolution technique when the transmission takes place over the BSC. These thresholds under sum-product 
decoding, which correspond to the asymptotic case where we let the block length tend to infinity, provide the 
indicated gaps to capacity in Table ITT] The value of A2 for each sequence is compared with the upper bound given 
in Theorem [4] Note that for calculating the bound in Theorem @] the parameter r introduced in (1231 ) satisfies 

e r = . 1 for the BSC whose crossover probability is equal to p, and we substitute here the threshold value 

V 4 p( 1 ~p) 

of p under the sum-product decoding. Also, for the calculation of this bound for such a BSC, Eq. (1511 ) gives that 
g\ = (1 — 2p) 2 . The average right degree of each sequence is also compared with the lower bound in Theorem Q] 
These comparisons show that for the considered sequences of LDPC ensembles, both of the theoretical bounds are 
fairly tight; the upper bound on A2 is within a factor of 1.3 from the actual value for the two sequences of LDPC 
ensembles while the lower bound on the average right degree is not lower than 83% of the corresponding actual 
values. As with the LDPC ensembles in Table H the ensembles referred to in Table HI1 were obtained by the density 



LDPC 


Gap to 




Lower bound 




Upper bound 


ense- 


capacity 




on Or 


A 2 


on A2 


mble 


(e) 




(Theorem [T} 




(Theorem |4j 


1 


1.85 ■ 


5.172 


4.301 


0.291 


0.371 


2 


6.18 • 10" a 


11.000 


9.670 


0.160 


0.185 



TABLE II 

Comparison of theoretical bounds and actual values of A 2 and a R for two sequences of LDPC ensembles 

TRANSMITTED OVER THE BSC. THE SEQUENCES ARE TAKEN FROM [27] AND ACHIEVE VANISHING BIT ERROR PROBABILITY UNDER 
ITERATIVE SUM-PRODUCT DECODING WITH THE INDICATED GAPS TO CAPACITY. 

evolution technique with the goal of minimizing the gap to capacity under a constraint on the maximal degree. 
Example 5: [On the fundamental system of cycles for capacity-approaching sequences of LDPC ensembles] 

Corollary |2] considers an arbitrary sequence of LDPC ensembles, specified by a pair of degree distributions, whose 
transmission takes place over an MBIOS channel. This corollary refers to the asymptotic case where we let the 
block length of the ensembles in this sequence tend to infinity and the bit error (or erasure) probability vanishes; 
the design rate of these ensembles is assumed to be a fraction 1 — e of the channel capacity (for an arbitrary 
e G (0, 1)). In Corollary [2j Eq. (fl~6l ) applies to a general MBIOS channel and a tightened version of this bound is 
given in ( fT71 ) for the BEC. Based on these results, the asymptotic average cardinality of the fundamental system 
of cycles for Tanner graphs representing codes from LDPC ensembles as above, where this average cardinality is 
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Gap (in rate) to capacity 



Fig. 3. Plot of the asymptotic lower bounds in Corollary [2] (see Eqs. d 1 6b and d!7t ) for three memoryless binary-input output- symmetric 
(MBIOS) channels. These lower bounds correspond to the average cardinality of the fundamental system of cycles for Tanner graphs 
representing codes from an arbitrary LDPC ensemble; the above quantity is normalized w.r.t. the block length of the ensemble, and the 
asymptotic result shown in this figure refers to the case where we consider a sequence of LDPC ensembles whose block lengths tend to 
infinity (according to the statement in Corollary [2] the degree distributions of the sequence of LDPC ensembles are assumed to be fixed 
and the block lengths of these ensembles tend to infinity). The bounds are plotted versus the achievable gap (in rate) between the channel 
capacity and the design rate of the LDPC ensembles. This figure shows the bounds for the binary symmetric channel (BSC), binary-input 
AWGN channel (BIAWGNC) and the binary erasure channel (BEC) where it is assumed that the design rate of the LDPC ensembles is fixed 
in all cases and is equal to one-half bit per channel use. 



normalized w.r.t. the block length, grows at least like log ~. We consider here the BSC, BEC, and BIAWGNC as 
three representatives of the class of MBIOS channels, and assume that the design rate of the LDPC ensembles is 
fixed to one-half bit per channel use. It is shown in Fig. [3] that for a given gap (e) to the channel capacity and 
for a fixed design rate, the extreme values of this lower bounds correspond to the BSC and BEC (which attain 
the maximal and minimal values, respectively). This observation is consistent with the last part of the statement in 
Corollary |2 



VI. Summary and Outlook 

This paper considers properties related to the degree distributions of capacity-approaching LDPC ensembles, and 
to the graphical complexity and the average cardinality of the fundamental system of cycles of their Tanner graphs. 
Universal information-theoretic bounds which are related to these properties are derived when the transmission of 
these LDPC ensembles takes place over an arbitrary memoryless binary-input output-symmetric (MBIOS) channel. 
The universality of the bounds derived in this paper stems from the fact that they do not depend on the full 
characterization of the LDPC ensembles but rather depend on the achievable gap between the channel capacity and 
the design rate of the ensemble. Some of these bounds are also expressed in terms of the bit error (or erasure) 
probability and the block length of the ensembles, and several other bounds refer to the asymptotic case where we 
let the block length tend to infinity and the bit error probability vanishes. The first category of the results introduced 
in this paper provides the following bounds which hold under maximum-likelihood (ML) decoding (and hence, 
they also hold under any sub-optimal decoding algorithm): 
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• Theorem Q] provides a lower bound on the average degree of the parity-check nodes of a binary linear block 
code where it is assumed that the code is represented by an arbitrary bipartite (Tanner) graph which corresponds 
to a full-rank parity-check matrix. A lower bound on the average right degree of the Tanner graph is related 
to the graphical complexity of the representation of a binary linear block code by such a bipartite graph, and 
also to the decoding complexity per iteration which is associated with a message-passing iterative decoder 
operating on this graph. The bound in Theorem Q] applies to finite-length binary linear block codes, and it is 
expressed in terms of the required bit error probability of the code under ML decoding or any other decoding 
algorithm. This theorem is adapted to hold for LDPC ensembles while relaxing the rather heavy requirement 
of full-rank parity-check matrices for codes from an LDPC ensemble, and addressing instead the design rate 
of this ensemble with its corresponding gap to capacity (see Discussion [2] in Section HVT ). The adaptation of 
Theorem Q] is also considered for randomly and intentionally punctured LDPC ensembles (see Discussion [3] in 
Section [TV]). One of the implications of this theorem is introduced in Corollary [2] which provides asymptotic 
lower bounds on the average cardinality of the fundamental system of cycles of LDPC ensembles in terms of 
the capacity of the communication channel and the achievable gap to capacity. All of these lower bounds grow 
like the logarithm of the inverse of this gap (in rate) to capacity (see also [9], [22], [23], [25]), and they provide 
quantitative measures of the graphical complexity and the average number of fundamental cycles of capacity- 
approaching LDPC ensembles in terms of the gap between the channel capacity and their design rate. These 
bounds demonstrate that the above quantities become unbounded as the gap to capacity vanishes, and they also 
provide a quantitative tradeoff between the performance of these ensembles, and the graphical complexity and 
average cardinality of the fundamental system of cycles of their Tanner graphs (which are important parameters 
for studying the performance and decoding complexity of iterative message-passing decoders). 

• Theorem [2] provides upper bounds on the degree distributions of capacity-approaching LDPC ensembles (both 
from the node and the edge perspectives), and it addresses the behavior of these degree distributions for any 
finite degree in terms of the achievable gap to capacity. We discuss the implication of this theorem later. 

The second category of results is specialized to iterative message-passing decoding algorithms (addressing in 
particular the sum-product decoding algorithm). The derivation of these results is based on the proofs of the above 
bounds combined with the stability condition. Theorems [3] and [4] provide upper bounds on the fraction of degree-2 
variable nodes and the fraction of the edges connected to these nodes for LDPC ensembles; these bounds are 
expressed in terms of the achievable gap to capacity of these ensembles under the sum-product decoding algorithm 
where it is assumed that the block length of these ensembles tends to infinity and the bit error (or erasure) probability 
vanishes. A byproduct of Theorems [3] and [4] and Lemma[5]is that while the fraction of degree-2 variable nodes stays 
positive (under some mild conditions) for capacity-approaching LDPC ensembles, the fraction of edges connected 
to degree-2 variable nodes vanishes, and it is upper bounded by the inverse of the logarithm of the achievable gap 
to capacity. The tightness of the bounds in Theorems [T|-@] is exemplified for some capacity-approaching LDPC 
ensembles under sum-product decoding (see Section [V]). These bounds are shown to be reasonably tight for general 
MBIOS channels, and are particularly tight for the binary erasure channel (BEC). 

It is interesting to note that there is a fundamental property which is in common and another property which 
distinguishes the behavior of the degree distribution of the variable nodes from the degree distribution of the parity- 
check nodes for capacity-approaching LDPC ensembles under iterative message-passing decoding. The property 
which is in common relies on Theorem Q] and its adaptation to LDPC ensembles in Discussion [2] First note that 
the behavior of the average left and right degrees is similar in the sense that their ratio depends on the design 
rate of the ensemble (so it depends on the channel capacity for sequences of capacity-achieving LDPC ensembles, 
and is positive as long as the channel capacity is less than 1 bit per channel use). According to Discussion [2j 
the average left and right degrees of an LDPC ensemble scale at least like the logarithm of the inverse of the 
gap (in rate) to capacity, and therefore become unbounded for capacity-achieving sequences of LDPC ensembles 
(even under ML decoding). The other property which distinguishes between the degree distributions of these two 
kinds of nodes is related to the fact that under some mild conditions, the fraction of degree-2 variable nodes 
stays strictly positive as the gap to capacity vanishes under the sum-product decoding algorithm (see Lemma [5] 
and Remark [9]) whereas, according to Theorem [2l the fraction of parity-check nodes of an arbitrary finite degree 
vanishes for capacity-achieving LDPC ensembles. This observation conforms with the behavior of the optimized 
pairs of the degree distributions of capacity-approaching LDPC ensembles for various MBIOS channels (see, e.g., 
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[27]). More explicitly, the degrees of the variable nodes, which are obtained numerically in [27] via the density 
evolution technique, span over a large range (i.e., the degrees of the variable nodes are distributed between 2 and 
the maximal allowed degree) whereas the parity-check nodes are almost right concentrated (i.e., the degrees of 
the parity-check nodes are almost fixed). This observation also conforms with the behavior of the right-regular 
capacity-achieving sequence of LDPC ensembles over the BEC [24]. 

In the following, we gather what we consider to be the most interesting open problems which are related to this 
research: 

• The asymptotic bounds in Corollary [2] address the average cardinality of the fundamental system of cycles for 
Tanner graphs representing LDPC ensembles where the results are directly linked to the average right degree of 
these ensembles. Further study of the possible link between the statistical properties of the degree distributions 
of capacity-approaching LDPC ensembles and some other graphical properties related to the cycles in the 
Tanner graphs of these ensembles is of interest. 

• The derivation of universal bounds on the number of iterations and the decoding complexity of code ensembles 
defined on graphs, measured in terms of the achievable gap (in rate) to capacity, is of theoretical and practical 
interest. In a currently ongoing work, this issue is addressed for the particular case of the BEC where we let 
the block length of these ensembles tend to infinity. 

• The lower bound on the conditional entropy which is given in ([8]) plays a key role in the derivation of 
Theorem Q] and accordingly, it is also crucial for the derivation of the other results in this paper. As opposed to 
this information-theoretic bound, the bounding techniques presented in [14] and [15] rely on statistical physics, 
and therefore do not provide a bound on the conditional entropy which is valid for every binary linear block 
code from the considered ensembles (for a survey paper which provides an introduction to codes defined on 
graphs and highlights connections with statistical physics, the reader is referred to [16]). It would be interesting 
to get some theory that unifies the information-theoretic and statistical physics approaches and provides bounds 
that are tight on the average and valid code by code. 

• Extension of the results in this paper to channels with memory (e.g., finite-state channels) is of interest. In 
this respect, the reader is referred to [9] which considers information-theoretic bounds on the achievable rates 
of LDPC ensembles for a class of finite-state channels. 
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